-
Notifications
You must be signed in to change notification settings - Fork 654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rare/Unusual Domain & Destination Flow Check #7807
Conversation
fixed two RareDestTrainig's methods. renamed bitmaps ( more concise with its functionality ). added serialization to redis: ( dumpRareDestToRedis and loadRareDestFromRedis methods )
Host extension with Rare Destination structures and methods
Host extension
* Edit suggestion * refactor: change position of curly braces --------- Co-authored-by: Leonardo Brugnano <[email protected]>
Rare dest: Destination hashing function available in epoch_feature
defined constant values instead of struct's values
refactored getDestinationHash serialization of training variables into redis refactor epoch feature code
Sync fork with latest dev updates
Just in case, I've already fixed the conflicts with base branch. Looking forward to move on with pr |
Please also check my comments inline in the code. Thank you. |
Thank you for the precious comments and suggestions, we're going to fix the code as soon as possible. We only have some doubts about the learning mechanism you suggested. The periodic training logic was implemented because we think that if a destination is not seen in a while then it makes sense to consider it rare again. If only a single initial training is done, it is not possible to do such thing, causing the bitmaps to fill up after some time, effectively making the check useless. In addition, we talked to @lucaderi some time ago, and he suggested that we proceed by implementing periodic training. Regarding the bitmaps in LocalHost, @lucaderi said that it was up to us to decide whether to do it for host or network interface. Anyway, we'll move the bitmaps to NetworkInterface as suggested. |
As of the periodic training, what I proposed is the easiest solution: just warn once for every rare destination detected, and ignore it for the future. Alternatively you can periodically rebuild the bitmaps, but this should happen while running the check (building a "shadow" bitmap and swapping the bitmaps after the learning period), without putting the check in "standby" during the learning periods as this is like disabling the check for several hours per day. |
A rare destination is as such if it has not been detected so far in any of the hosts currently monitored. The learning period (or grace period) is a way to avoid triggering alerts until some basic knowledge is made. So it's done once at beginning and not periodically. Instead periodically you need to keep a backlog of visited hosts that allow you to trigger events for sites visited seldom (i.e. once a week). Does all this make sense to you? |
new implementation of Redis save/load minor bug fixes & typo modular get/set methods for rareDestTraining values LocalHost cleanup
deallocating chain in case of failre
@DropB1t Anything we can do to expedite this PR? |
As suggested by the review: - Status ( bitmaps ) has been decoupled: for domain & IPs - Status has been moved into NetworkInterface, and now it's global for each interface - Inline comments fixes has been made - Learning mechanism has been streamlined
As suggested by the review:
|
My comments after the last review:
|
I hope I have explained myself well. Thank you for your attention and time |
Closing for inactivity |
Me and my colleagues @LeoBubi & @F3rto have implemented a check Flow Check for rare domains and destinations, using
ndpi_bitmap
, with periodic learning.Mentioned issues are
#6416, #6417